Sure, but that's an argument about scale not about fundamental limitations of the architecture. The OP is saying the LLM approach can't be intelligent. Not eve

Sure, but that's an argument about scale not about fundamental limitations of the architecture. The OP is saying the LLM approach can't be intelligent. Not even if you add more parameters, more RAM, more scale. You're saying that an LLM might be able to do what you can do if we make the matrices bigger. That's a different claim. Of course, the real trick, and I think it's clear this is the missing step, is that LLM training can read a thousand page book and embed that knowledge into the weights. So what you want to do is take some of the prompt input, and use it to update the weights in the model (or the LORA, or the fine-tuning, or whatever), not just the context. That's not how LLMs are working today. It's not clear to me that we're so many steps away from that being something we can do though.