3 ms·

I'm not sure why you say these claims are unrelated. The article claims to have found the existence of concepts embedded in the trained parameters of a model.

I'm not sure why you say these claims are unrelated. The article claims to have found the existence of concepts embedded in the trained parameters of a model. (You called this having a "systemic understanding of the rules.") You are saying that for this to be so, the model would have to perform with 100% accuracy. The general form of this argument is that for a system of intelligence to be endowed with a conceptual understanding of a ruleset, it would have to implement that ruleset with 100% accuracy, right? Well, given that human's don't implement rulesets with 100% accuracy, we'd then need to concede that either 100% consistency is not a requirement for understanding or else human's do not operate with understanding.

nickelpro 4y ago
Exactly, the claim that humans do not operate with perfect understanding is far more interesting (and IMHO, likely) than the idea that LLMs are spontaneously producing accurate models of the systems used to generate their data. And again, the claims are unrelated. There's no reason to believe that humans and LLMs operate under similar principles, and a ton of reasons to believe that they absolutely do not.
- crosen99 4y ago
  This is what you said originally: "We only need a single counter example to show that Othello-GPT does not have a systemic understanding of the rules, only a statistical inference of them." I don't believe this statement to be true. It is clearly not true that a single counter example rules out systemic understanding of rules for ALL intelligent systems. Human intelligence is offered as the case in point. So, given that the statement is not true for all systems, by what logic do you conclude that the statement is true specifically for an LLM?
  - nickelpro 4y ago
    > It is clearly not true that a single counter example rules out systemic understanding of rules for ALL intelligent systems. I didn't say anything about "intelligence", the OP is not about "intelligence". The OP claim is, "we find interesting evidence that simple sequence prediction can lead to the formation of a world model". A world model, an accurate one, is a very different thing than a statistical model. An inaccurate world model is indistinguishable from a statistical model. An accurate world model, definitionally, does not make mistakes. These models are trivial to produce, classical chess engines are accurate world models. > Human intelligence is offered as the case in point. I think you've hit on a more interesting point about human intelligence, that it is often not an accurate world model, which means it is inaccurate or more likely, statistical in nature. > So, given that the statement is not true for all systems, by what logic do you conclude that the statement is true specifically for an LLM? So what we're left with is that OthelloGPT (and humans, but again, what's going on with humans is irrelevant to the study of an LLM) is in one of two scenarios, it either: A) Has an inaccurate world model, or we might say, an accurate world model of a game that is not Othello B) Is still basically just a curve fitting algorithm, which there's all the evidence in the world to suggest it is It sits upon the authors to distinguish between these two choices, and they never offer any evidence either way. So I side with B.

nickelpro 4y ago

Exactly, the claim that humans do not operate with perfect understanding is far more interesting (and IMHO, likely) than the idea that LLMs are spontaneously producing accurate models of the systems used to generate their data. And again, the claims are unrelated. There's no reason to believe that humans and LLMs operate under similar principles, and a ton of reasons to believe that they absolutely do not.

crosen99 4y ago
This is what you said originally: "We only need a single counter example to show that Othello-GPT does not have a systemic understanding of the rules, only a statistical inference of them." I don't believe this statement to be true. It is clearly not true that a single counter example rules out systemic understanding of rules for ALL intelligent systems. Human intelligence is offered as the case in point. So, given that the statement is not true for all systems, by what logic do you conclude that the statement is true specifically for an LLM?
- nickelpro 4y ago
  > It is clearly not true that a single counter example rules out systemic understanding of rules for ALL intelligent systems. I didn't say anything about "intelligence", the OP is not about "intelligence". The OP claim is, "we find interesting evidence that simple sequence prediction can lead to the formation of a world model". A world model, an accurate one, is a very different thing than a statistical model. An inaccurate world model is indistinguishable from a statistical model. An accurate world model, definitionally, does not make mistakes. These models are trivial to produce, classical chess engines are accurate world models. > Human intelligence is offered as the case in point. I think you've hit on a more interesting point about human intelligence, that it is often not an accurate world model, which means it is inaccurate or more likely, statistical in nature. > So, given that the statement is not true for all systems, by what logic do you conclude that the statement is true specifically for an LLM? So what we're left with is that OthelloGPT (and humans, but again, what's going on with humans is irrelevant to the study of an LLM) is in one of two scenarios, it either: A) Has an inaccurate world model, or we might say, an accurate world model of a game that is not Othello B) Is still basically just a curve fitting algorithm, which there's all the evidence in the world to suggest it is It sits upon the authors to distinguish between these two choices, and they never offer any evidence either way. So I side with B.