The problem with LLMs, which are really impressive, is that they have too many parameters. This makes them fragile, as they lack regularization. IMHO, a very i

The problem with LLMs, which are really impressive, is that they have too many parameters. This makes them fragile, as they lack regularization. IMHO, a very interesting research route is to combine small mechanistic models with big networks such as LLMs, where the latter play the role of intuition. Research by Tenenbaum from MIT, and other similar groups, is heading in this direction.