I have the same feeling. I often wonder about the difference between learning and reasoning, in neural networks. If we could train a small network on a small am

I have the same feeling. I often wonder about the difference between learning and reasoning, in neural networks. If we could train a small network on a small amount of code and it figures out how to write code that's as good as what GPT-4 can write, that's what I'd call reasoning (and that's what's currently a massive open research question). On the other hand, it's definitely a lot easier to just show a gigantic model a gigantic amount of code until it becomes familiar with most types of applications a user would likely ask it about. So it feels like right now we're papering over our lack of research success by (essentially) throwing money at the problem, and it seems to be working well enough. (At least, well enough to be useful to people.)