3 ms·

> Obviously there are limits to what a small model is capable of doing This is not obvious to me. I mean yes, limits exist, but I seriously doubt that we are a

> Obviously there are limits to what a small model is capable of doing This is not obvious to me. I mean yes, limits exist, but I seriously doubt that we are anywhere close to them. This is exactly the kind of question that needs academic research! I feel that our current models waste tons of compute and memory and there ought to be room to optimize them by 10x or even much more with new techniques.

jszymborski 3y ago
By leaning on inductive biases, we've historically been able to get pretty far with relatively small models and litte data. The trouble is, applying inductive biases requires that you make assumptions about the problem, which requires that you understand the problem. The stronger the inductive bias, the stronger the assumptions. Creating fantasticly large models allows us to eschew the inductive bias in favour of model capacity. That capacity would lead to useless, fragile models which overfit without mountains of data. It's great to explore the maxima of either approach so we can define some kind of loose Pareto frontier of model size and performance. I do think there's tonnes of room to benefit from the pissing contest the tech giants are currently engaged in and learn from the massive models what can be done for smaller models. The RAG pipeline is kind of an approach to add some inductive bias to such large models. Currently people tend to use the same giant causal LLMs for RAG as they do for non-RAG Chat purposes, but I'm certain you can get much smaller models performing very well for RAG. In fact, I'm sure plenty of folks are doing it already considering the number of eyeballs on LLM and RAG atm.
wavemode 3y ago
I have the same feeling. I often wonder about the difference between learning and reasoning, in neural networks. If we could train a small network on a small amount of code and it figures out how to write code that's as good as what GPT-4 can write, that's what I'd call reasoning (and that's what's currently a massive open research question). On the other hand, it's definitely a lot easier to just show a gigantic model a gigantic amount of code until it becomes familiar with most types of applications a user would likely ask it about. So it feels like right now we're papering over our lack of research success by (essentially) throwing money at the problem, and it seems to be working well enough. (At least, well enough to be useful to people.)

jszymborski 3y ago

By leaning on inductive biases, we've historically been able to get pretty far with relatively small models and litte data. The trouble is, applying inductive biases requires that you make assumptions about the problem, which requires that you understand the problem. The stronger the inductive bias, the stronger the assumptions. Creating fantasticly large models allows us to eschew the inductive bias in favour of model capacity. That capacity would lead to useless, fragile models which overfit without mountains of data. It's great to explore the maxima of either approach so we can define some kind of loose Pareto frontier of model size and performance. I do think there's tonnes of room to benefit from the pissing contest the tech giants are currently engaged in and learn from the massive models what can be done for smaller models. The RAG pipeline is kind of an approach to add some inductive bias to such large models. Currently people tend to use the same giant causal LLMs for RAG as they do for non-RAG Chat purposes, but I'm certain you can get much smaller models performing very well for RAG. In fact, I'm sure plenty of folks are doing it already considering the number of eyeballs on LLM and RAG atm.

wavemode 3y ago

I have the same feeling. I often wonder about the difference between learning and reasoning, in neural networks. If we could train a small network on a small amount of code and it figures out how to write code that's as good as what GPT-4 can write, that's what I'd call reasoning (and that's what's currently a massive open research question). On the other hand, it's definitely a lot easier to just show a gigantic model a gigantic amount of code until it becomes familiar with most types of applications a user would likely ask it about. So it feels like right now we're papering over our lack of research success by (essentially) throwing money at the problem, and it seems to be working well enough. (At least, well enough to be useful to people.)