>I think the short of it is that it seems to me that you are confusing a system having very good heuristics for having a solid understanding of principles of re

>I think the short of it is that it seems to me that you are confusing a system having very good heuristics for having a solid understanding of principles of reality. Humans don’t have a grasp of the “principles of reasoning” and as such are incapable of distinguishing “true”, "different" or “heuristic” assuming such a distinction is even meaningful. Where you are convinced of “faulty shortcut”, I simply think “different”. Multiple ways to skin a cat. a plane's flight is as "true" as any bird. There's no "faulty shortcut" even when it fails in ways a bird will not. You say humans are "true" and LLMs are not but you base it on factors that can be probed in humans as well so to me, your argument simply falls apart. This is where our divide stems from. >I don't think you've provided evidence that AI can have a principled understanding while we can show that humans can. What would be evidence to you? Let’s leave conjecture and assumptions. What evaluation exist that demonstrate this “principled understanding” in humans? and how would we create an equitable test in LLMs? >a novice would be able to identify that somebody is drawing a circle around it's pieces; your link even states this. But you recharacterizing this as a specific strategy is weird when that strategy causes you to lose the game. You misunderstand. I did not characterize this as a specific “strategy”. Not only do modern Go systems not learn like humans, but they also don’t learn from human data at all. KataGo didn’t create a heuristic to play like a human because it didn’t even see humans play. >The paper you referenced similarly shows it is reliant on proximity to the training set, rather than actually understanding the underlying principles. Even the authors make it clear this isn’t necessarily the bridge to take so it’s odd to see you die on this hill. The counterfactual of syntax is Finding the main subject and verb of something like “Think are the best LMs they.” in verb-obj-subj order (they, think) instead of “They think LMs are the best.” in subj-verb-obj order (they, think). LLMs are not being trained on text like the former to any significant degree if at all yet the performance is fairly close. So what, it doesn’t “underlying principles of syntax” but still manages that ? The problem is that you take a fairly reasonable conclusion from these experiments. I.e LLMs can/often also rely on narrow, non-transferable procedures for task-solving and proceed to jump the shark from there. >but you haven't convinced me that it can fundamentally change away from it's training set based on an understanding of basic principles. We see language models create novel functioning protein structure after training, no folding necessary. https://www.researchgate.net/publication/367453911_Large_language_models_generate_functional_protein_sequences_across_diverse_families https://www.researchgate.net/publication/367453911_Large_lan... So does it still not understand the “basic principles of protein structures”?